Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



liiillliiilililiii 

(11) EP 0 701 369 A1 



(12) EUROPEAN PATENT APPLICATION 

(43) Date of publication: (51) |nt 01.6; H04N 7/01 

13.03.1996 Bulletin 1996/11 

(21) Application number: 95306297.3 



(22) Date of filing: 08.09.1995 



(84) Designated Contracting States: 


• Matsugu, Masakazu 


DE ES Fn GB IT NL 


Shimomaruko, Ohta-ku, Tokyo (JP) 




• Yano, Kotaro 


(30) Priority: 09.09.1994 JP 216323/94 


Shimomaruko, Ohta-ku, Tokyo (JP) 


21.06.1995 JP 154654/95 


• Kurahashi, Sunao 


30.06.1995 JP 166233/95 


Shimomaruko, Ohta-ku, Tokyo (JP) 




• Kondo, Toshiaki 


(71) Applicant: CANON KABUSHIKI KAISHA 


Shimomaruko, Ohta-ku, Tokyo (JP) 


Tokyo (JP) 


• Mori, Katsuhiko 




Shimomaruko, Ohta-ku, Tokyo (JP) 


(72) Inventors: 


• Ishlkawa, Motohiro 


• lijima, Katsumi 


Shimomaruko, Ohta-ku, Tokyo (JP) 


ShimomarukOi Ohta-ku, Tokyo (JP) 




• Tokunfiltsu, Jun 


(74) Representative: 


Shimomaruko, Ohta-ku, Tokyo (JP) 


Beresford, Keith Denis Lewis et al 




BERESFORD & Co. 




2-5 Warwick Court 




High Hoiborn 




London WC1R 5DJ (GB) 



(54) Image processing method and apparatus 



(57) An image sensing apparatus according to an 
embodinnent of the present invention includes an innage 
input unit (100) for inputting an image, a detection unit 
(200) for detecting a state of movement of the image in- 
put unit (1 00) in an Image input operation, a storage unit 
(310, 9000) for storing a plurality of images input by the 
image input unit (100) and movement information corre- 
sponding to the state of movement detected by the de- 
tection unit (200), and an Image generating unit (320) for 
generating an image from an arbitrary viewpoint position 
on the basis of the plurality of images and the movement 
information stored in the storage unit. 
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Description 

BACKGROUND OF THE INVENTION 
FIELD OF THE INVENTION 

The present invention relates to an image processing nnethod and apparatus which sense an object (to be sensed) 
by using a plurality of image sensing systems or a single Image sensing system and reproduce a three- or two-dimen- 
sional image from the sensed image. 

DESCRIPTION OF THE RELATED ART 



Distance measuring devices for measuring the distance to an object to be photographed can be roughly classified 
into distance measuring devices based on active methods of irradiating some kind of energy onto the object and distance 
IS nrieasuring devices based on passive methods of analyzing an image signal sensed by a photoelectric conversion ele- 
ment or the like. 

Active methods which are widely used are: a method of irradiating a wave, e.g., a radio wave, an ultrasonic wave, 
or light, onto an object and measuring the distance to the object on the basis of the propagation time required for the 
Irradiated wave to return upon reflection; and a method (e.g., Moir6 topography) of obtaining the shape of an object to 
20 be photographed from the geometric distortion of an artificial pattem formed on the object surface by projecting a regular 
pattern from an appropriate light source. 

Passive methods are: a method of measuring a distance, on the basis of the principle of trigonometrical survey, 
from the positional relationship between corresponding points of images obtained by using a plurality of image sensors; 
and a method of obtaining the three-dimensional structure of an object to be photographed from the geometric distortion 
2S of a pattern on the object surface. In addition, attempts have been made to use a method of obtaining the distance to 
an object on the basis of images sensed by a plurality of Image sensing processing systems having different focal lengths. 

As described above, in various conventional methods, the three-dimensional shape of an object is measured to 
extract three-dimensional information, and a three-dimensional image Is reproduced on the basis of the three-dimen- 
sional Information. 

30 In general, in order to reproduce images, sensed Image data and images viewed from predetermined viewpoints 

and formed by interpolation on the basis of sensed data are stored in advance. In reproducing images, images are 
selected from the stored images to be displayed. 

Several problems are posed in realizing the above various problems. 

First of all, there are problems associated with extraction of three-dimensional information required to generate a 
35 reproduced Image. More specifically, In the above conventional methods, since energy such as an ultrasonic wave or 
a laser beam must be irradiated on an object to detect the reflecting object, measurement cannot be performed for any 
object which absorbs the energy. In addition, a strong laser beam cannot be Irradiated on a human body in consideration 
of adverse effects on the human body. That is, the above active methods are poor In versatility. Furthermore, the method 
of projecting a regular pattern is basically a method used to obtain the relative three-dimensional information of an object 
40 so that an absolute distance until the object can not be acquired. 

Of the passive methods, the method of using a plurality of image sensors requires a plurality of image sensing 
elements and image sensing systems and hence is not suitable for a compact, inexpensive distance measuring device. 
In the method of extracting three-dimensional information from the geometric distortion of a pattern on an object surface, 
the characteristics of the geometric pattern must be known in advance, and obtained information is only relative distance 
45 information. For this reason, the application range of this method is very narrow. In addition, a distance measuring device 
based on a focus error with respect to an image requires a bright lens with a small depth of focus and a large numerical 
aperture to perform accurate distance measurement. For this reason, sufficient resolution cannot be obtained by using 
a general TV camera lens. 

The above problems are posed in measuring the three-dimensional shape of an object or an object to be photo- 
so graphed. Secondly there is no conventional apparatus which reproduces an image viewed from an arbitrary viewpoint 
position and an arbitrary viewpoint direction from images obtained by sensing an object to be photographed. For exam- 
ple, a conventional video apparatus can only reproduce an image under the same condition as that in a recording 
operation. This will be described with reference to Fig. 1 . When a photographing operation is performed by a general 
camera, an image is obtained in accordance with the TV frame (field) frequency, as shown in Fig. 1 . In reproducing this 
55 image, forward reproduction Is performed from the position denoted by reference numeral 181 in Fig. 1, or reverse 
reproduction is performed from the position denoted by reference numeral 182. An image reproduction order cannot be 
set in accordance with a designated order based on an arbitrary viewpoint position and an arbitrary position of line of sight. 
As described above, since many Image data including many interpolated Image data must be held to reproduce an 
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image, a large-capacity memory is needed. In addition, a much retrieval time is required to pick up one of image data 
which has been stored in the memory. Owing to the limited memory capacity, it is not practical to retain all images viewed 
from various viewpoints. 

5 SUMMARY OF THE INVENTION 

The present invention has been made to solve the above problem, and has as a concern to provide an image 
sensing/reproducing apparatus which can easily obtain a three- or two-dimensional image viewed from an arbitrary 
viewpoint and an arbitrary viewpoint direction. 
10 It is another concern of the present invention to provide an image processing method and apparatus in which the 

storage capacity of a recording unit which is required to reproduce images viewed from various viewpoint directions is 
small. 

It is still another concern to provide an image processing method and apparatus which can accurately obtain the 
information of the distance to an object to be photographed without irradiating special energy onto the object. 
IS It is still another concern to provide an image processing method and apparatus which can easily obtain the moving 

direction and amount of an image sensing apparatus and accurately reproduce an image on the basis of the obtained 
information. 

In accordance with the invention there is provided an image sensing apparatus comprising: image input means for 
inputting an image; detection means for detecting a state of movement of said image input means in an image input 

20 operation; storage means for storing a plurality of images input by said image input means and movement information 
corresponding to the state of movement detected by said detection means; and image generating means for generating 
an image from an arbitrary viewpoint position on the basis of the plurality of images and the movement information 
stored in said storage means. 

Another aspect of the invention provides an image sensing method comprising: image input step of inputting an 

25 image using image Input means; detection step of detecting a state of movement of said image input means in an image 
input operation; storage step of storing a plurality of images input at said image input step and movement information 
corresponding to the state of movement detected by said detection step; and image generating step of generating an 
image from an arbitrary viewpoint position on the basis of the plurality of images and the movement information stored 
in said storage step. 

30 Other features and advantages of the present invention will be apparent from the following description taken in 

conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts 
throughout the figures thereof. 

BRIEF DESCRIPTION OF THE DRAWINGS 

35 

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embod- 
iment of the invention and, together with the description, sen/e to explain the principles of the invention. 

Fig. 1 is a view showing a conventional image sensing method and a conventional reproduction method; 

40 

Fig. 2 is a block diagram showing the arrangement of an image processing apparatus according an embodiment of 
the present invention; 

Fig. 3 is a block diagram showing the arrangement of an image processing apparatus according to the first embod- 
"^5 iment of the present invention; 

Fig. 4 is a view showing an example of the arrangement of a multi-eye image sensing unit; 

Fig. 5 is a view showing how an image sensing operation is performed; 

50 

Fig. 6 is a view showing the relationship between the position of each image photographed in accordance with the 
photographing sequence in Fig. 5 and the time; 

Fig. 7 is a view showing the concept of image interpolation; 

55 

Fig. 8 is a block diagram showing an image interpolation unit; 
Fig. 9 is a view for explaining corresponding points extraction; 
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Fig. 10 is a view schematically showing a state wherein an object point P is projected on an image sensing sensor; 
Figs. 11 A and 11 B are views for explaining a method of performing interpolation by using spatio-temporal images; 
5 Fig. 1 2 is a view showing images after interpolation; 

Fig. 1 3 is a view showing the reproduction order of images; 

Fig. 1 4 is a view showing the reproduction order of motion images; 

10 

Fig. 15 is a flow chart showing a sequence of image processing in the first embodiment; 

Fig. 1 6 is a view illustrating interpolation areas when a photographing operation is performed by a multi-eye image 
sensing unit; 

IS 

Fig. 17 is a block diagram showing a processing arrangement for reproducing a three-dimensional image at an 
arbitrary viewpoint in an arbitrary viewpoint direction; 

Fig. 18 is a view for explaining a designated viewpoint direction; 

20 

Fig. 19 is a block diagram showing a processing arrangement for performing image reproduction when an image 
input unit moves back and forth; 

Fig. 20 is a view for explaining processing to be performed when the image input unit moves back and forth; 

25 

Fig. 21 is a block diagram showing an image processing arrangement using a multi-eye image sensing unit; 

Fig. 22 is a view showing the relationship between the photographing range of images sensed by the image process- 
ing arrangement in Fig. 21 and the time; 

30 

Fig. 23 is a block diagram showing a processing an^angement for extracting distance information of an object to be 

photographed; 

Fig. 24 is a block diagram showing a processing arrangement for generating a reproduced image from designated 
35 viewpoint/viewpoint directions in accordance with distance information; 

Fig. 25 Is a view showing the relationship between photographed images and a designated reproduction order; 

Fig. 26 is a view for explaining processing to be performed until images are reproduced in accordance with a des- 
40 ignated reproduction order; 

Figs. 27 A and 27B are views showing the relationship between the threshold for determining identical points and 
the moving amount; 

45 Figs. 28A and 288 are views for explaining processing for obtaining distance information; 

Fig. 29 is a flow chart showing a sequence of processing of generating/synthesizing distance information; 

Fig. 30 is a block diagram showing an image processing arrangement having a multi-eye image sensing unit and 
so a corresponding points input unit; 

Fig. 31 is a view showing how corresponding points are input; 

Fig. 32 is a block diagram showing the arrangement of a main processing unit according to the eighth embodiment 
ss of the present invention; 

Fig. 33 is a view showing the relationship between images obtained before and after a multi-eye image sensing unit 
moves; and 
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Fig. 34 is a block diagram showing the arrangement of a corresponding-points input unit. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

s Preferred embodiments of the present invention will be described in detail in accordance with accompanying draw- 

ings. 

The hardware arrangement of an image processing apparatus according to an embodiment of the present invention 
will be briefly described with reference to Fig. 2. 

Reference numeral 2005 denotes an image sensing unit for generating image data by sensing an object to be 
10 photographed. The generated image data is processed when a MPU (Microprocessor Unit) 2000 sequentially executes 
various image processing programs (to be described later) stored in a ROM 2002. A RAM 2001 is used as a working 
area required for this processing. 

The image data input from the image sensing unit 2005 is stored in an image memory 2006. Image data generated 
by interpolation processing (to be described later) is also stored in the image memory 2006. 
IS A display l/F (InterFace) 2003 performs control to display the image data as a result of the above processing based 

on various image processing programs on a display 2004. 

An MM l/F (Man-Machine InterFace) 2007 performs data input control of various input devices for inputting various 
command/data, e.g., a pen 2011 , a mouse 2010, a joy stick 2009, and a polhemus sensor (trademark registration by 
Polhemus Corp.) 2008. 

20 

[First Embodiment] 

The arrangement of a single-eye Image sensing system as one form of the image sensing unit of the present in- 
vention will be mainly described next with reference to Fig. 3. 

25 Blocks 1 00 and 200 respectively indicated by the broken lines are an image input unit and a moving amount detection 

unit for the image input unit 100. These blocks 100 and 200 constitute the image sensing unit 2005 in Fig. 2. The block 
100 is an image input unit for inputting an image data from an object (not shown). Reference numeral 110 denotes an 
image sensing lens for sensing an object to be measured; 1 20, an image sensing element such as a CCD; 1 30, a sample 
hold (S/H) circuit for holding an output signal from the image sensing element (CCD) 120; 140, automatic gain control 

30 (AGC) circuit; and 150, an analog/digital (A/D) converter. 

The block 200 is a moving amount detection unit for detecting the moving amount and direction of the image input 
unit 100. Reference numeral 210 denotes an angular velocity sensor such as a vibration gyroscope; and 220, a moving 
amount calculation circuit for calculating the moving amount and direction of the image input unit 100 from the angular 
velocity detected by the angular velocity sensor 21 0. 

35 The angular velocity sensor 21 0 is designed to perform detection In three axial directions, i.e., the x-. y-, and z-axis 

directions. The moving amounts output from the moving amount calculation unit 220 can be output as moving amounts 
Bx, By, and Bz along the x-, y-, and z-axes. Alternatively, the overall moving amount can be output, as indicated by: 

B = 7bx^ +By^ + Bz^ 

40 The moving amount output from the moving amount detection unit 200 is buffered in a memory 9000 of a main 

processing unit 320, and is output from the main processing unit 320 as needed. 
Note that the memory 9000 is assigned to a RAM 2001 . 

A block 300 indicated by the broken line is an image processing unit for calculating the distance to an object to be 
measured by causing the respective units other than the image sensing unit 2005 In Fig. 2 to execute image signal 
45 processing. Reference numeral 310 denotes an image memory assigned to a partial area of the image memory 2006 
in Fig. 2 and used to hold image data for a predetermined period of time; and 302, the main processing unit for image 

processing. 

The main processing unit 320 inputs viewpoint locus data 330 and generates images corresponding tothe respective 
viewpoints from image data sensed at time n and time n-1 and input from the A/D converter 150 and an image memory 
so 310. 

The viewpoint locus data 330 is input through various data input devices connected to the MM l/F 2007. 

Note that processing performed by the main processing unit 320 will be described in detail later. 

The image sensing lens 110 forms an image of the object on the image sensing element 120. The image sensing 
element 120 photoelectrically converts the formed image of the object. The S/H circuit 130 holds the output from the 
S5 image sensing element 120. The AGC circuit 140 then controls the gain of an Image signal. The AJD converter 150 
analog/digital-converts the output from the AGC circuit 140, and outputs the resultant digital image as Image data. 

The angular velocity sensor 210 detects the movement of the image input unit 100 itself. The moving amount cal- 
culation unit 220 calculates the moving amount of the image input unit 100 from the detected angular velocity of the 
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image input unit 100, and outputs it to the image processing unit 300. This moving amount corresponds to a baseline 
length in a stereoscopic display. 

The image memory 310 stores the image data from the image input unit 100 for a predetermined period of time and 
allows the main processing unit 320 to perform calculation between images in a time serial order. 
s The arrangement shown In Fig. 3 is constituted by the single-eye Image sensing system. However, the arrangement 

may be constituted by a multi-eye image sensing systems. An embodiment using this multi-eye image sensing unit will 
be described below with reference to Fig. 4. 

Referring to Fig. 4, the multi-eye image sensing unit of this embodiment is designed to sense an object to be pho- 
tographed by using two image sensing systems, i.e., right and left image sensing systems 10r and IOl. 
10 The left image sensing system IOl will be described first. The left image sensing system 10|_ includes an image 

sensing lens group 1 1 l as an Imaging optical system incorporated In a lens barrel (not shown), a color separation prism 
12|_ attached to the image sensing lens group 11j_ to separate light from an object to be photographed into light beams 
of three primary colors, and three CCD sensors 1 3l (only one is shown) respectively having rectangular effective light-re- 
ceiving portions and serving as image sensing elements arranged for the respective light beams of the colors separated 
IS by the color separation prism 1 2l. 

The image sensing lens group 11|_ is constituted by a focusing lens group 15l driven by a focus motor 14l, and a 
plurality of lenses including a various magnification lens group driven by a zoom motor 16l. The motors 14l and 
16l are driven in accordance with control signals from a total system control unit 21 and a focus/zoom control unit 22 
in a control unit 20 for controlling the optical system. The right image sensing system IOr has the same arrangement 
20 as that of the left Image sensing system 10l. An optical axis Lr of an image sensing lens group 11 r of the right image 
sensing system lOp is on the same plane as that of an optical axis Ll of the Image sensing lens group 11l of the left 
image sensing system IOl. 

The lens barrels incorporating the image sensing lens groups 1 1 l and 11 r are respectively coupled to the rotating 
shafts of convergence angle motors 1 8l and 1 Br driven in accordance with control signals from a convergence angle 

25 control unit 23. The rotating shafts of the convergence angle motors 1 8l and 1 8r extend in a direction perpendicular to 
a plane Including the optical axes Ll and Lr of the Image sensing lens groups 11 l and 11 r. When the convergence 
angle motors 18l and 18r are driven, the image sensing lens groups 11 l and 11 r are rotated together with the color 
separation prisms 1 2l and 1 2r and the CCD sensors 1 3l and 1 3r to set an angle (convergence angle) defined by the 
optical axes Ll and Lr of the Image sensing lens groups 1 1l and 11 r. 

30 The right Image sensing systems IOl ^Or respectively include focus encoders 24l and 24r for detecting the 

positions of the focusing lens groups 1 5l and 1 5r, zoom encoders 25l and 25r for detecting the positions of the various 
magnification lens groups 1 and 1 7r, and convergence angle encoders 26l and 26r for detecting convergence angles. 
These components may be components to be externally mounted, such as potentiometers, or components for detecting 
the positions and angles of the lens groups from signal information of driving systems, such as pulse motors, themselves. 

35 An Image processing unit 301 performs image processing similar to that perfomned by the main processing unit 320 

(in Fig. 3). 

An image output unit 40 serves to display image data processed/generated by the Image processing unit 301 on 
the display 2004 (In Fig. 2) through the display l/F 2003 (in Fig. 2) and store processed/generated Image data In the 
Image memory 2006. 

40 An Image sensing method using an innage sensing unit having the same arrangement as that in Fig. 3 or 4 will be 

described next. 

Assume that a photographing operation Is performed in front of a cage in a zoo while an image sensing apparatus 
(camera) is moved in the direction indicated by an arrow 31 , as shown in Fig. 5. Fig. 6 shows the relationship between 
the photographing position or the time of photography and the range of a sensed image in a case wherein the Image 
45 sensing unit in Fig. 3 is used. Referring to Fig. 6, the abscissa corresponds to the range of a sensed image (the size of 
an object surface); and the ordinate, time. Reference numeral 41 in Fig. 6 denotes the range of a sensed image. 

In this photographing operation, an Image 42, i.e., an image viewed from a virtual viewpoint, is Interpolated from 
the time of photography between images 41 , as shown in Fig. 7. Fig. 8 shows the concept of this processing. 

"Image input" In Fig. 8 is an input through the A/D converter 1 50 in Fig. 3. Assume that the current time of photography 
so Is n. In this case, image data at time n-1 has been input to the Image memory 31 0, and the images at time n and time 
n-1 are simultaneously input to an image interpolation processing unit 322 constituting part of processing performed by 
the main processing unit 320 (Fig. 3). That is, the above operation is equivalent to input of a stereo image. 

Interpolation processing to be performed by the image interpolation processing unit 322 is executed In synchronism 
with input of the respective viewpoint coordinates of the viewpoint locus data 330 described above. 
55 As is apparent, the viewpoint locus data 330 can include the above virtual viewpoint positions. 

Interpolation processing methods in the Image interpolation processing unit 322 will be described next. 

The interpolation methods include the following two methods: 
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(1 ) a method of extracting Information of corresponding points of two or more images, and performing interpolation 
by using a distance calculated on the basis of trigonometrical survey by using the extraction result and the photo- 
graphing parameters of the camera; and 

(2) a method of performing interpolation by using spatio-temporal epipolar plane Images. 

The method (1 ) will be described first. In order to extract a pair of corresponding points, for example, a block matching 
method is used. According to this method, for example, a block enclosing a certain point in a left image is considered, 
and similarities in a right image corresponding to the image in the block are compared to determine corresponding points. 
In a correlation method as one of the methods of comparing similarities, the cross-correlation between a pixel value In 
the image in the block and a pixel value in a image to be retrieved is calculated, and a representative point in the block 
which exhibits the maximum value is determined as a corresponding point. This relationship is represented by the fol- 
lowing equation (1): 

CT(mR, HR, mL, ni) 



Z R(mR - i, HR - j) • L(mL + riL + j) 
i> j 



X R^dTiR - i, riR - j) • I X R2(jnL + i, hl -f j) 



. . . (1) 



where a is the normalized cross-correlation value. 

In equation (1), RCm^.n^) and MmL.nJ are the pixel values of the left and right images, a(mR, n^, mL, nj is the 
correlation degree, and (mR.np) and (m|_,nL) are the pixel coordinates In the orthogonal X and Y coordinate spaces. 
Note that in the square sum or product sum term in equation (1), opposite signs are set before "i" and "j" corresponding 
to the left and right images because pixel coordinate axes are defined such that the left and right images become 
symmetrical, as shown in Fig. 9. In normalized cross-correlation in equation (1), the maximum value is 1. 

As another interpolation method, an SSDA method is available, which is a kind of block matching. 

CT (mR, hr, mL, riL) 

= Z Z |R(mR - riR - j) - L(mL + i, hl -h j)| . . . (2) 
i j 

In this method, a residual a is calculated according to equation (2). If the residual exceeds the certain threshold, which 
is set in advance, in the process of addition, the calculation is stopped, and then the next combination of (mR,nR) and 
(mL,n|_) is processed. In general, the threshold Is set to the minimum residual in the past. 

The positions of the respective corresponding points in the three-dimensional space are obtained from the obtained 
corresponding point information by the trigonometrical sun/ey method. As shown in Fig. 10, central points 0|_ and Or 
of the object-side major surfaces of the left and right image sensing lens groups 11l and lip (see Fig. 4) are arranged 
on the X-axis to be symmetrical about the Z-axis, and the length of a baseline connecting the central points Ol and Or 
is represented by b. In this case, the coordinates of the central points Ol and Or are respectively represented by 
(-b/2,0,0) and (b/2.0,0). In addition, when a point P in the three-dimensional space is projected toward the central points 
Ol and Or, projection points on the left and right CCD sensors 1 3l and 1 3r are respectively represented by Pl and Pr. 
The coordinates of the points P, Pl. and Pr are respectively represented by (X,Y,Z), (Xl,Yl,Zl), and (Xr,Yr,Zr). In this 
case, a plane defined by the points P, Pl, and Pr is called an epipolar plane, and the line of intersection between the 
epipolar plane and the sensor plane is called an epipolar line. 

The coordinates (X,Y,Z) of the point P are given by the following equations: 

{X, +(b/2)}/Z, +{Xp-(b/2)}/Z„ 
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Let e be the angles (convergence angles) defined by the optical axes Ll and Lr of the left and right image sensing 
lens groups 1 1 l and 1 1 r, which pass through the central points Ol and Or of the object-side major surfaces, respectively, 
and straight lines parallel to the Z-axis, and f be the focal lengths of the image sensing lens groups 1 1 1^ and 1 1 r, then 

^ Zr = {Xr - (b/2) + f.sin(e) }tan(0) + f .cos(G) 

Zl = -{Xl + (b/2) - f .sin(e) }tan(G) + f.cos(0) (6) 

According to the above equations, the coordinates (X,Y,Z) of the point P can be obtained. An image viewed from a 
certain viewpoint, e.g., an image viewed from an intermediate point 0(0,0,0) between the two Image sensing systems, 
is obtained by coordinate transformation processing on the basis of these coordinates. 

The method (2), which uses a spatio-temporal Image, will be described next. As shown in Fig. 11 A, a point P is 
projected onto points e, f, g, and h when the camera is placed at positions A, B, C, and D. These points and positions 
are arranged vertically, as shown in Fig. 11B. In this case, it seems as if the points P were arranged in a line. Assume 
that a straight line passes through the points e to h. In this case, if a position A', which Is shown by a dotted line, is 
assumed between the positions A and B, an Intersection A becomes an interpolated pixel. 

According to this method, an interpolated image is obtained by interpolating a line in epipolar images and substituting 
the pixel value of a corresponding point for a pixel corresponding to the point of intersection between the line and a 
detected straight line. 

As is apparent, the above problem of corresponding point extraction is replaced with the problem of straight line 
^ detection to be solved. In addition, an omission of part of a straight line, i.e., an occlusion portion, can be interpolated 
by connecting with a straight line. 

By using these method of interpolating an image based on a virtual viewpoint, images 42 which are shown by dotted 
lines, as shown in, e.g.. Fig. 12, can be generated. 

The interpolated images are stored in the image memory 2006. 

A method of 330 will be described next with reference to Fig. 1 3. Fig. 13 shows an image, which is for inputting the 
viewpoint locus data displayed on the display 2004, for example. In an initial state of the image, the abscissa representing 
the photographing range and the ordinate representing the time of photography are displayed. 

In this case, the user sequentially inputs the central positions (corresponding to virtual viewpoints) of images to be 
reproduced by using various data input devices connected to the MM l/F 2007. 

In the case shown in Fig. 13, when a first point 110 is input, points 111, 112, and 1 1 3 are sequentially input. At this 
time, the dotted line indicated by the points 110, 111 , 112, and 113 indicates the reproductbn order. These point coor- 
dinate data strings are the viewpoint locus data 330 described above. 

Referring to Fig. 13, one dotted line is plotted from an upper right position to an upper left position. Also, the line 
can be arbitrarily drawn in the form of, e.g., a loop or the figure "8". Therefore, the degree of freedom greatly increases 
^ in terms of reproduction order as compared with the conventional fonward reproduction and backward reproduction. 

As described above, interpolated images generated in correspondence with the viewpoint locus data 330 and ac- 
tually sensed images are sequentially read out from the image memory 2006 and displayed on the display 2004. 

The method of generating an interpolated image when an image is viewed from an arbitrary viewpoint has been 
described above, together with the method of reproducing generated images in an aitoitrary order. A case wherein an 
^ object to be photographed is moving will be described next. 

Assume that the camera moves in the direction indicated by an arrow 1 30 in Fig. 1 4. In this case, the image sensing 
range is an area 1 31 between the two broken lines. Assume that a hippopotamus as an object to be photographed is 
moving in the direction Indicated by an arrow 132. In this case, areas 133 and 134 in Fig. 14 are respectively an area 
in which interpolation may be performed as if the hippopotamus was present and an area which can be interpolated as 
^ an area where the hippopotamus was not present. 

Interpolation processing in this case will be described below. The movement of the camera can be measured with 
the moving amount calculation unit 220 in Fig. 3 or the like. The movement of the object, i.e., the hippopotamus, is 
sensed as the sum of the movement of the camera and the movement of the object. 

Assume that the object moves slowly like the hippopotamus. In this case, if a photographing operation is performed 
at the TV field frequency, since the change from a given frame to the next frame is small, the interpolation error is also 
small. In the above interpolation method using a spatio-temporal image, since an image Is interpolated upon detection 
of a straight line, omissions due to occlusion and the like can be interpolated. 

In this case, if a reproduction order 1 35 is input, an image string can be reproduced in the following order: a back- 
ground image without the hippopotamus a portion having no image an image with the hippopotamus moving in 
different directions. In general, a photographing operation is performed while the camera is picking up an object. That 
is, a photographing operation is rarely performed in the above manner. In addition, reproduction can be performed in 
such a manner that the photographed object moves backward or moves fonward in various directions, e.g., oblique 
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directions. If, therefore, such a reproduction technique is applied to motion Image processing especially in the field of 
animation, an Improvement can be attained in terms of entertainment. 

The overall image processing in the image processing apparatus of this embodiment will be described next with 
reference to the flow chart of Fig. 15. Note that the programs corresponding to this flow chart are stored in the ROM 
2002 in advance, and are sequentially read out by the MPU 2000 to be Interpreted and executed. 

In step S1. image data sensed by the Image sensing unit 2005 at the respective discrete times are input to the 
image memory 2006. 

In step S2, the flow waits for input of a viewpoint position serving as a request sent from the MM (Man-Machine 
Interface) l/F 2007 to request display of a reproduced image viewed from a desired viewpoint at a desired time point. If 
the viewpoint position is input, the flow advances to step S3. 

In step S3, an image is interpolated by the method described In detail above on the basis of the viewpoint position 
input in step S2 and the sensed image data stored in the image memory 2006, and the interpolated image is stored In 
the image memory 2006. 

In step S4, the interpolated image is displayed on the display 2004 through the display l/F 2003 on the basis of the 
interpolated image generated in step S3. The flow then returns to step S2 to wait for viewpoint data to be Input next, 
and the same processing as described above is repeated. 

[Second Embodiment] 

The image processing method of the first embodiment has been mainly described with reference to the single-eye 
iniage sensing unit (image input unit 100) in Fig. 32. 

The image processing method of the second embodiment will be mainly described with reference to a case wherein 
image data is input by using a multi-eye image sensing unit (Fig. 4). 

Fig. 16 shows a state wherein a photographing operation is performed at a multi-eye Image sensing system con- 
vergence angle 0. Reference numeral 140 denotes an optical axis set when a photographing operation is performed by 
using a single-eye image sensing system; 141 and 142, left and right optical axes set when a photographing operation 
is performed by using the multi-eye image sensing system; and 1 30, the locus of a camera in a case wherein a photo- 
graphing operation is performed by using the single-eye image sensing system. Assume that the multi-eye image sensing 
system moves along the same locus as the locus 130 of the single-eye Image sensing system. 

An area Aofh (144) and an area Aaic (145) are interpolation areas when the single-eye image sensing system is 
used. An area Aefg (143) and an area Abed (146) are interpolation areas when the multi-eye image sensing system is 
used. An interval hi (131) is a photographing range when the single-eye image sensing system is used. An interval gi 
(147) is a photographing range when the multi-eye image sensing system is used. As Is apparent from Fig. 16, the 
photographing range (1 47) is larger than that set when the single-eye image sensing system is used. Consequently, the 
range In which interpolation can be performed is widened. Note that in this case, interpolation and reproduction methods 
are the same as those in the first embodiment. 

[Third Embodiment] 

The third embodiment exemplifies the processing method of three-dimensionally displaying an image reproduced 
by using an interpolated image generated in the first or second embodiment. Assume that in this embodiment, the 
multi-eye image sensing unit in Fig. 4 is used as an image input means. 

Fig. 17 schematically shows a sequence of reproduction processing. Referring to Fig. 17, two images input from 
the image sensing unit 2005 at a given time, i.e., an L (left) image and an R (right) image, are subjected to panoramic 
synthesis in a panoramic synthesizing unit 400. Note that a panoramic synthesis method is disclosed in detail in Japanese 
Patent Application No. 5-223544 filed by the present applicant on September 8, 1993. This image having undergone 
panoramic synthesis is stored in an image memory 401 , and an Interpolated image is generated by an interpolated 
image generating unit 402 in the same manner as described above. L (left) images and R (right) Images are stored in 
Image memory areas 403 and 404 respectively. These image memory areas 401. 403, and 404 are assigned to the 
image memory 2006. 

The interpolated image generating unit 402 performs basically the same processing as that in the image interpolation 
processing unit 322 described above except that a panoramic image is input. 

Data to be input from a reproduction order generating unit 406 Is the same as the viewpoint locus data 330. That 
is, the viewpoint locus data 330 is generated by a reproduction order input unit 405 and the reproduction order generating 
unit 406. In this case, the reproduction order input unit 405 corresponds to the respective data input devices connected 
to the MM l/F 2007. The reproduction order generating unit 406 converts the viewpoint position data input through the 
reproduction order input unit 405 into data in a predetermined internal digital form, generates a corresponding input 
string, and sequentially outputs the data of the input string to the interpolated image generating unit 402. 
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The L and R images respectively stored in the image memory areas 403 and 404 are transferred to transformation 
units 407 and 408. 

In synchronism with the timing at which each data of the viewpoint locus data 330 described above is input to the 
interpolated image generating unit 402, the interpolated image generating unit 402 generates interpolated panoramic 
stereo images corresponding to the above data from the input panoramic images at time n-1 and time n, i.e., stereo 
panoramic images, and stores the generated images in the image memory areas 403 and 404. 

A viewpoint direction input unit 409 inputs a viewpoint direction corresponding to each designated viewpoint direction 
input through the reproduction order input unit 405. The transformation units 407 and 408 perform affine transformation 
of stereo images input from the image memory areas 403 and 404 on the basis of this viewpoint direction. 

Fig. 1 8 shows an example of how this viewpoint direction data Is input. Similar to Fig. 1 3, the abscissa represents 
the photographing range; and the ordinate, the time of photography. 

This image is displayed on the display 2004. The user can designate and input a viewpoint direction as the direction 
of an arrow, together with a desired stereo image position (viewpoint) at which the image is to be reproduced, by using 
the data input devices connected to the MM l/F 2007. This viewpoint direction input processing is performed by the 
viewpoint direction input unit 409. 

Referring to Fig. 17, the processing units other than the image memory areas 401 , 403, and 404 and the display 
2004 are constituted by software, and the corresponding programs are stored in the ROM 2002 in advance. These 
programs are sequentially read out, interpreted, and executed by the MPU 2000. 

A transformation processing methods in the transformation units 407 and 408 will be described next. 

This transformation is realized by calculating the following matrix: 
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... (7) 

Where X, Y, and 2 are the coordinates of an object point P; A, B. and C are the rotational angles corresponding to 
viewpoint directions at arbitrary viewpoint positions, and X*, Y', and Z' are the coordinates of a point P' to which the 
object point P has moved in accordance with a change in viewpoint direction. 

The transformation units 407 and 408 receive these rotational angles from the viewpoint direction input unit 409, 
and calculate the coordinates (X, Y,Z) of the respective points of the stereo images input from the image menriory areas 
403 and 404 according to the expression (7), thereby acquiring the coordinates (X'.Y'.Z') of the respective points corre- 
sponding to the designated viewpoint direction. 

The stereo images transformed in this manner are displayed on the display 2004. 
. As described above, stereo images can be reproduced at arbitrary viewpoint positions, in arbitrary viewpoint direc- 
tbns, and in an arbitrary reproduction order. 
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The fourth embodiment further includes an image transformation unit like the one shown in Fig. 19 to cope with a 
case wherein a viewpoint moves back and forth. This embodiment uses the following transformation method. As shown 
in Fig. 20, object points a and b are projected as points a' and b' at the position of an image sensing sensor 170. When 
the sensor moves to the position of a sensor 1 71 by a moving amount v, the points a and b are sensed as points a" and 
b". The points a" and b" can be obtained by simple proportional calculations from a focal length f, the moving amount 
V, and the coordinates of the points a' and b'. 

Referring to Fig. 1 9, image data 1 60 are stereo Image data having undergone transformation in the transformation 
units 407 and 408 or stored in the image memory areas 403 and 404. These data are input to a transformation unit 161 
to perform image transformation according to the above calculations in accordance with the movement of the viewpoint 
in the back-and-forth direction. 

A back-and-forth position input unit 162 receives the moving amount v from various data input devices connected 
to the MM l/F 2007, and outputs it to the transformation unit 161. 

Note that a sequence (program) of processing in the transformation unit 161 and the back-and-forth position input 
unit 162 is stored in the ROM 2002 in advance, and is executed by the MPU 2000. 

The above image processing method and apparatus of the present invention can be applied to three-dimensional 
shape measuring devices for obtaining three-dimensional information from two-dimensional images, such as an envi- 
ronmental recognition apparatus, an obstacle recognition apparatus, a geographical shape measuring device, and a 
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remote sensing apparatus, which are mounted in an unmanned traveling vehicle, a self-propelled robot, and the like, 
and an object or environmental basic shape Input device required for a solid model such as a CG or CAD. In addition, 
a multi-or single-eye image sensing apparatus having a simple three-dimensional measuring device can be applied to 
an outdoor video recording/reproducing apparatus which is carried by the user, e.g., to a zoo, to photograph animals 
and sceneries, and reproduce recorded information on the spot or perform reproduction at home to enjoy Images at 
viewpoints different from those in the recording operation with different effects. 

As described above, the above embodiment comprises the image input means, the image storage means, the 
means for detecting the moving direction and amount of the image input means, the image interpolation means, the 
Image reproduction means, the means for inputting a reproduction order, and the viewpoint position/viewpoint direction 
input means. With this arrangement, three- or two-dimensional Images of still and moving objects viewed from arbitrary 
viewpoint positions and arbitrary viewpoint directions can be reproduced in an arbitrary reproduction order. 

In the above embodiment, an image sensing operation is performed, while the single-eye camera is moved, to form 
a stereo image. However, in the fifth embodiment, two single-eye cameras are used to acquire a stereo image, and 
image processing is performed. 

[Fifth Embodiment] 

Fig. 21 is a block diagram showing the arrangement of a three-dimensional shape measuring device according to 
the present invention. 

The arrangement shown in Fig. 21 is basically obtained by adding an image input unit 101 having the same ar- 
rangement as that of the image input unit 100 to the arrangement shown in Fig. 3. 

A description of the arrangements of a moving amount detection unit 200 and the image input units 100 and 101 
will be omitted, and a different processing unit, i.e., an image processing unit 350, will be mainly described below. 

Viewpoint locus data 330 and viewpoint direction data 331 input to a main processing unit 360 are identical to those 
input to the reproduction order Input unit 405 and the viewpoint direction input unit 409 described above with reference 
to Fig. 17. 

An image memory 370 in the image processing unit 350 simultaneously receive image data respectively output 
from the image input units 100 and 101, and store the data for a predetermined period of time, thereby allowing the 
main processing unit 360 to perform an operation between a given image and the next image as time series data. A 
processing unit 320 performs various image processing operations like in the fifth embodiment. For example, the image 
processing unit 350 (except the image memory 370) is constituted by software, and the corresponding processing pro- 
grams are stored in the ROM 2002. These programs are executed by the MRU 2000. 

Fig. 22 shows the photographing range and the time of photography of image data 41 and 410 to be input to the 
main processing unit 360. 

Fig. 22 also shows a state wherein the user holds a unit Including the image input units 1 00 and 1 01 and the moving 
amount detection unit 200 with his/her hands in front of a cage in a zoo, and performs a photographing operation while 
moving in the direction indicated by the arrow 31 in Fig. 5. 

Referring to Fig. 22, the abscissa represents the range of sensed images; and the ordinate, the time base. Intervals 
41 and 410 in Fig. 22 respectively represent the ranges of images sensed by a multi-eye Image sensing unit as in this 
embodiment. 

Assume that the apparatus Is moved in the direction indicated by an arrow 31 from time t^, to time t^ , and the multi-eye 
cameras 100 and 101 shift parallel. Also, assume that the moving amount calculated by a moving amount calculation 
unit 220 is a moving amount Bz. This information Is sent to the main processing unit 360. The main processing unit 360 
performs processing, e.g., obtaining an object distance Z^^^. 

Image data output from the image memory 370 are stereo Images respectively output from the Image input units 
100 and 101 at time n-1 with respect to the Images 41 and 410 at time n. For the sake of easy understanding, Fig. 22 
does not show such data. 

Fig. 23 is a block diagram for explaining the detailed arrangement of part of the main processing unit 360. 

Referring to Fig. 23, the above moving amount B or the above moving vector (Bx.By.Bz) are Input to an object 
distance coupling unit 500 of the main processing unit 360. 

The object distance coupling unit 500 adds an object distance (depth) Z^-, calculated by a distance calculation unit 
502 (to be described later) to an output Bj, from the moving amount calculation unit 220. Therefore, with reference to 
the position of each camera at time Xq, a calibrated object distance Is given by: 

Zi,i'=Ziti+Bz 

Similarly, Bx and By are added to and Yn-, respectively output from the distance calculation unit 502 to obtain 
calibrated object distances Xj^' and Y■^^^\ The number of calibrated object distances obtained in this manner increases 
with time t, thereby detailed distance data in a wide range can be obtained by performing a photographing operation a 
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plurality of times in the direction indicated by tlie arrow 31, as shown in Fig. 22. These distance data are sent to an 
image generating unit 801 (hto be described later). 

The contents of processing performed by the distance calculation unit 502 will be described next. The stereo image 
data 41 and 410 corresponding to discrete times t^, t^,..., t^ in Fig. 22 are sequentially input to the distance calculation 
5 unit 502, which calculates three-dimensional information corresponding to the input order, and outputs the resultant 
object distances X^j, Yj^, and Z■^^ described above to the object distance coupling unit 500. In order to acquire three-di- 
mensional information, the distance calculation unit 502 extracts corresponding points between the stereo image data 
41 and 410 input at each time. The distance calculation unit 502 obtains the three-dimensional information of the object 
by using the corresponding points and photographing parameters for the cameras, e.g., focal lengths f and a distance 
10 b between the two cameras on the basis of the principle of trigonometrical survey. 

This corresponding point extraction method has already been described in the first embodiment, and hence a de- 
scription thereof will be omitted. 

Assume that the obtained corresponding points in the Images 41 and 410 which correspond to an object point Pi 
at time to have coordinates (VL,hL) and (vR.hp). In this case, an object distance Zjjo with respect to the object point Pi is 
IS given by: 

Z -f ^ 

where f is the focal length and b is a given baseline length between the two cameras 100 and 101. These values are 
20 Image sensing parameters. 

I n this manner, all object distances Zj are obtained from a plurality of corresponding points obtained from the Images 
41 and 410. 

Note that 

25 Y V -7 ^ito 

^itO - f ' ^ itO - '^jtO f 

The image processing unit as a characteristic feature of this embodiment will be described next with reference to 
Fig. 24. Fig. 24 shows the concept of image reproduction processing. The arrangement shown in Fig. 24 constitutes 
part of the main processing unit 360 (Fig. 21 ). 

30 Upon reception of one viewpoint position data of the viewpoint locus data 330, an image selection unit 800 selects 

two pairs of stereo images nearest to the viewpoint position. This operation will be described below with reference to 
Fig. 25. Fig. 25 shows an image which is for inputting an image reproduction order, displayed on the display 2004 when 
an image reproduction order is to be input. The photographing range of the stereo imaged data 41 and 410 which have 
already been sensed are displayed on the display 2004 in correspondence with the time of photography. The user 

35 designates a viewpoint string indicating a reproduction order by using various data input devices connected to the MM 
l/F 2007. The designated viewpoint string data is input, as the viewpoint locus data 330, to the image selection unit 800. 
The image selection unit 800 selects two pairs of stereo images nearest to each viewpoint of the viewpoint string, and 
sends first selected stereo images (71 L (Left) and 71 R (Right)) and second selected stereo images (72L and 72R) to 
the image generating unit 801. 

40 The image generating unit 801 includes an image interpolating unit 811 as an interpolating means for interpolating 

the difference between images, and an image rotating unit 810 for rotating an image in accordance with the viewpoint 
direction data 331 . 

The image interpolating unit 811 performs interpolation of an image corresponding to a viewpoint designated by the 
viewpoint locus data 330 on the basis of the two pairs of stereo images (71 L and 71 R) and (72L and 72R). This inter- 
ns polation method has already been described above, and hence a description thereof will be omitted. For example, a 
technique associated with this interpolation is disclosed in Japanese Patent Application No. 6-216323. The image ro- 
tating unit 81 0 then performs rotational transformation of the image obtained by the interpolation in accordance with a 
viewpoint direction designated by the viewpoint direction data 331. 

For this rotational transformation, a calculation may be performed in accordance with matrix (7), as described above. 
so The stereo images (72 L and 72R) having undergone rotational transfomiation are stored in the image memory 2006, 

and are displayed on the display 2004 through the display l/F 2003. 

As described above, according to this embodiment, not only images actually sensed in a photographing operation 
can be reproduced, but also Images can be sequentially reproduced as images viewed from an arbitrary direction in 
accordance with a designated reproduction order. 
ss In addition, since designated images are sequentially interpolated in accordance with distance information in a 

designated reproduction order, a large storage capacity is not required. 

A state of processing to be performed until images are output in the above designated reproduction order will be 
described with reference to Fig. 26. 
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Fig. 26 explains processing to be performed to reproduce corresponding images in accordance with a designed 
reproduction order 710. 

First of all, as indicated by T26A" in Fig. 26, the images 41 and 410 are sequentially input and stored in the image 
memory 2006 in accordance with a movement 31 in a photographing operation. The reproduction order 710 is then 
input, as described above. 

As reproduced images at a point 1210 In the reproduction order 7 10, images 70L, 70R, 71 L, and 71 Ron both sides 
of an Intersection between the movement 31 and a vertical line extending from the point 1210 are selected. The selected 
images are indicated by '^268" in Fig. 26. The images 70L and 70R are viewed from the two sides of a cube as an 
object to be photographed. The images 71 L and 71 R are viewed from a position shifted to the right from the Images 
70L and 70R by a moving amount 111. 

Images at the above point of Intersection are interpolated from these four Images 70L, 70R, 71 L, and 71 R, and are 
transformed into images viewed from the viewpoint by using distance information. As a result, images 72L and 72R are 
generated (T26C"). These two images are displayed on the display 2004, for example, such as an HMD (Head-Mounted 
Display). 

[Sixth Embodiment] 

The sixth embodiment exemplifies the corresponding point extraction method which attains a higher accuracy than 
the corresponding point extraction method in the first embodiment by also taking the correspondence between images 
in the time base directbn Into consideration. 

First of all, the above-described block matching is performed between left and right Images as stereo images sensed 
at time to, and the obtained corresponding points are expressed as: 

(^itO' ^itO' ^ito)- ('="•» 2, ...n; corresponding point number) 

The corresponding points obtained by block matching at time t^ are expressed as: 

(X^ . Y^^ , Z^^^ ): (k = 1, 2,.... n; corresponding point number) 

In this case, an evaluation value E representing the similarity between corresponding points of the point numbers i and 
k Is calculated according to the following equation: 

E = {Xjto - (Xkti + B^)f + {Y,^ - (Y^, + By) f + {Z^^ - (Z^, + B^}f (8) 

In this case, the points (Xjto, Yj^Q, Zjto) and (X^ti , Y^ti , Z^^ ) which satisfy E < th(B) are considered as identical points. 
At this time, this threshold th(B) Is changed In accordance with, e.g., a moving amount B. 



More specifically, when the moving amount B (= 7bx^ + By^ + Bz^) is small, the threshold th(B) Is set to be a small 
value. For example, th = a»B (a is a positive coefficient) (Fig. 8A). 

If the moving amount B is large, the threshold th(B) is set to be large. By determining the threshold th(B) adaptively 
in this manner, identical point determination can be accurately performed. 

If the moving amount B and the like can be detected at a high resolution, the following equation can be established, 
as shown in Fig. 8B: 

B > Bo 
C -pB 0 < B < Bo 



th = 



... (8) 

C +pB -Bo < B < 0 

D B < -Bo 



Note that D = C - pg^ and p is a predetermined coefficient. 

With this operation, when a photographing operation is performed from a near place, identical point determination 
Is performed relatively loosely, whereas when a photographing operation Is performed from a distant place, identical 
point determination is performed strictly. 

The above description is associated with the images sensed at times Xq and t^. As is apparent, however, the same 
identical point determination processing as described above is performed with respect to images at times t^^ and t^+i (n 
= 0, 1...., m). 

As described above, in addition to distance information obtained from corresponding points of left and right Images, 
the distance information of the corresponding object point P to be sensed can be obtained by using corresponding points 
in the time base direction which are obtained in this embodiment. Therefore, the accuracy of the distance infomnation 
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of the object point P can be increased. 
[Seventh Embodiment] 

In the seventh embodiment, a method of obtaining distance infomiation on the basis of the relationship between a 
moving amount B and a baseline length b to further Increase the accuracy of distance information associated with 
corresponding points will be described with reference to Figs. 28A, 28B, and 29, 

The processing in this embodiment will be briefly described first with reference to Figs. 28A and 28B. 

As shown in Fig. 28A, distance information Z^q (i = 0, 1 n; corresponding point number) from two images L900 

and R910 sensed at time to on the basis of the above block matching and trigonometrical survey. Distance information 
Zkti (kO, 1,.... n; corresponding point number) is then obtained from two images L901 and R911 sensed at time t^. The 
sixth embodiment exemplifies the method of obtaining corresponding points between images in the time base direction 
on the basis of the pieces of distance information Z^o and Z^ji. obtained at two times to and t^. In contrast to this, in the 
seventh embodiment, as shown in Fig. 28B, Z,j,i and Z^^ are sequentially obtained respectively regarding left Images 
at times to and t^ and right images at times to and t-, . This procedure is explained below. First of all, Z^o is obtained from 
two images L920 and R930 sensed at time to- Distance Information Z^^ Is then obtained from two images L921 and 
R931 sensed at time t^ . I n addition to the two distance information Zj^o and Z^^ obtained in this order, distance information 
Z„i is obtained from the images L920 and L921 respectively sensed at times tO and t^, and distance information Z^^ Is 
obtained from the images R930 and R931 respectively sensed at times to and t^. 

In this case, 7^^ and 7^1 are obtained from information obtained when the same image sensing apparatus moves 
by the moving amount B. 

Fig. 29 is a flow chart showing the process of generating/synthesizing these pieces of distance information. 

In step S11 , a distance Z^^ at the ith corresponding point is calculated on the basis of left and right images 920 and 
930 respectively sensed at time to- 1 n step SI 2, the image sensing apparatuses (1 00 and 1 01 ) are moved by the moving 
amount B, and left and right images 921 and 931 are sensed. Similar to step S1 1 . in step SI 3, a distance Zy^ at the kth 
corresponding point is calculated. In step S14, distances Z,t-, and Z^i at the corresponding points are obtained on the 
basis of the calculation results in steps S11 and SI 2 and equation (8). 

In step S1 5, it is checked whether the moving amount B output from a moving amount calculation unit 220 is smaller 
than the baseline length b of the multi-eye camera. If the moving amount B is smaller than the baseline length b, the 
flow advances to step SI 7 to store the distances Zj^o and Zj^^ as the distance information of the corresponding points 
In the image memory 2006 without considering the distances Z„i and Z^^. If the moving amount B is larger than the 
baseline length b, the flow advances to step S17 to store the four distance information Z|to, Z^^, Z,„, and Z^^ as the 
distance information of the corresponding points In the image memory 2006. 

The processing of the images at times to and t^ has been described above with reference to the fbw chart of Fig. 
29. As is apparent, however, images at times t^ and t^^^ are processed in the same manner as described above. 

As described above, according to this embodiment, the distance information of an object to be photographed which 
is required to interpolate an image at an arbitrary viewpoint position in an arbitrary viewpoint direction can be extracted 
at a high resolution. Therefore, high-resolution interpolation/reproduction can be performed. 

[Eighth Embodiment] 

According to the eight embodiment, there is provided an image processing method and apparatus which include 
an Input means for designating/inputting a plurality of corresponding points with respect to sensed stereo images and 
in which representative movement vectors for an image input unit are obtained on the basis of a plurality of corresponding 
input points, and interpolated Images are generated by using the representative movement vectors and displayed. 

In this embodiment, since accurate representative movement vectors can be obtained, image reproduction can be 
performed at a higher resolution. 

Referring to Fig. 30, blocks 100 and 101 indicated by the broken lines are image input units for sensing an object 
to be measured to obtain image data. Since the above blocks 100 and 101 are identical to those in Fig. 21, the same 
reference numerals in Fig. 30 denote the same parts as in Fig. 21. 

Stereo images sensed by the image input units 100 and 1 01 are Input to an image processing unit 800. The Image 
selection unit 800 performs interpolation processing of images and the like on the basis of corresponding points input 
from an Image rotating unit 810, which is a characteristic feature of this embodiment, the moving amounts/directions 
(i.e., movement vectors) input from a moving amount calculation unit 820, reproduction order data, and viewpoint direc- 
tion data. 

The corresponding point input unit 810 includes a display device 811 and a pointing device 812. The display device 
81 1 displays the stereo images input from the image input units 1 00 and 1 01 . The pointing device 81 2 is used to input 
data for designating the positions of corresponding points on the stereo images displayed on the display device 811. 
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Upon reception of this position designation data, the display device 811 displays a predetermined mark at each corre- 
sponding displayed image position. The input corresponding point position designation data is sent to the moving amount 
calculation unit 820. 

The processing performed by the corresponding point Input unit 810 will be described next with reference to Fig. 
31 . Fig. 31 shows a state wherein two stereo images are displayed on the monitor of the display device 81 1 . 

On the upper portion of the display device 811, first and second images 8310 and 8311 as images at time t are 
displayed. 

On the lower portion of the display device 811 , third and fourth images 8320 and 8321 as images at time t+At are 
displayed. The first image 8310 is an image obtained by the Image input unit 100 at time t. The third Image 8320 Is an 
image obtained by the image input unit 100 at time At. 

The user inputs the positions of corresponding points on the stereo images with the pointing device 812 while 
watching this screen. In the case shown in Fig. 31, points 3201, 3202, 3203, 3204. and 3205 are made to correspond 
to points 3101, 3102, 3103, 3104, and 3105. 

The images displayed on the display device 811 described above based on image data stored In an Image memory 
1310. 

A method of generating a movement vector (Bx.By.Bz) using the moving amount calculation unit 820 will be de- 
scribed next. 

Assume that the image input units 100 and 101 move to (Bx,By,Bz) in a time interval At. and rotate to the ((d,0,%) 
direction. 

Fig. 33 shows the coordinate system of images and the state of movement In this case. Assume that images 4000 
and 4001 are image sensing planes. Consider a coordinate system (x,,y|,Z|) In which the x-axis and the y-axis are re- 
spectively parallel to the U| axis and the Vj axis with a camera principal object Ol sen/ing as the origin. Also, assume 
that the central position of the image 4000 is set to (0,0,0) and the posture of the image input unit 100 is set to (0,0,0). 

In this case, the position of the image input section 100 for sensing an Image 4001 Is set to (Bx,By,B2), and the 
posture is set to ((o,0,%). Letting Bx be the unit length, 
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where by By/Bx, b^ = B^/B^, and f is the focal length of the image input units 100 and 101. 

In the above determinantal equation, (by.b^) and (o),0,x) are determined to satisfy the following determinantal equa- 
tion and the relationship between a pair of corresponding point coordinate values Input from the corresponding point 
input unit 810: 
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55 



That is, In order to determine five unknown values, five or nnore corresponding point coordinate pairs (u€, v€) and 
(u€', v€') are input. With substitution of these values into the above equations, the same number of simultaneous equa- 
tions as the number of input pairs can be obtained. These simultaneous equations are treated for the minimization of 
the square sum of the obtained equation errors to identify the unknown values (by.bx) and ((0,0,%). That is, the above 
unknown values are identified by using the least squares method. 

A method of calculating the movement vector (B^, By.B^) normalized by a baseline length B on the basis of the values 
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(by.bjj) and (o.o.x) identified in this manner will be described below. 

First of all, In order to normalize the obtained moving amounts by and b^ by the baseline length B between the image 
input units 100 and 101, a distance z to a corresponding object point is obtained by using the points 3101, 3111, and 
3201 designated as corresponding points on the first, second, and third images 8310, 8311 , and 8320. 

For the sake of descriptive convenience, assume that the Image Input units 1 00 and 1 01 are arranged parallel, and 
are moved parallel. In this case, the distance z is obtained from the points 3101 and 3111 , as follows: 

z = fB/d 

where d is the difference between the points 3101 and 3111 on the image in the u^ coordinate system. 
Also, a distance z' is obtained from the points 3101 and 3201 as follows: 

z'=fB,/d' 

where d' is the difference between the points 3101 and 3201 on the image in the U| coordinate system. 
In this case, since z = z', = d'/dB. Therefore, a relative moving amount by is given by: 

by = By/Bx 

= By/ (d'/dB) 
Similarly, a relative moving amount b^ is given by: 

bx = Bz/Bx 

= B^/ (d'/dB) 

The normalized movement vector (Bx,By,B2) is given by the following determlnantal equation: 
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The movement vector (B^^.By^^) obtained according to the above sequence of processing and corresponding to 
each Image sensing time point can be sent to a main processing unit 1 320. 

The processing arrangement of the main processing unit 1320 will be described next with reference to Fig. 32. Fig. 
32 shows the detailed processing arrangement of the main processing unit 1320. 

The processing arrangement of the main processing unit 1320 will be briefly described first. 

Stereo images sensed by the image input units 1 00 and 1 01 are input to a distance processing unit 520. The distance 
to the object point corresponding to each pixel Is obtained on the basis of the principle of trigonometrical survey or the 
like In consideration of the movement vector (Bjj,By,B2) input from the moving amount calculation unit 820. The calculation 
result is output to an Image transformation unit 3608 and an image interpolatton unit 3605. 

The distance processing unit 520 is constituted by a distance calculation unit 510 and an object distance coupling 
unit 500. This arrangement will be described In detail later. 

The image interpolatton unit 3605 performs image interpolation processing on the basis of reproduction order data 
representing the arrangement of the respective viewpoint positions required for image reproduction, corresponding point 
data input from the corresponding point input unit 810, distance data input from the object distance coupling unit 500, 
and image data stored in an image memory 1 31 0. The interpolated image is sent to the image transformation unit 3608. 

The image transformation unit 3608 performs rotational transformation of the interpolated image on the basis of 
viewpoint direction data. An image reproduction unit 3606 sends the resultant image to the display l/F 2003 to display 
the image on the display 2004. 

A sequence of processing performed by the distance processing unit 520 will be described next. The distance 
processing unit 520 performs processing for calculating the object distance. 

First of all, the distance calculation unit 510 calculates a distance from corresponding points on stereo images and 
a given baseline length B by trigonometrical survey without considering the above movement vector. 

In this case, corresponding points can be obtained in three ways. The first corresponding point acquisition method 
is the acquisition method based on block matching described in the first embodiment. The second corresponding point 
acquisition method is a method of obtaining a distance on the basis of corresponding points input through the corre- 
sponding point input unit. Let Zj^ be the distance to the Ith object point obtained at time t. The third corresponding point 
acquisition method is a method of using both a block matching operation and a corresponding point input operation. 
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More specifically, input corresponding points are directfy determined as corresponding points, and corresponding points 
obtained by the block matching method are used as the remaining points which are not input. 

The object distance Zj, calculated by the distance calculation unit 51 0 Is input to the object distance coupling unit 500. 

The object distance coupling unit 500 obtains the corrected true object distance from the input object distance Zjj 
5 and the movement vector (Bjj.By.B^) input from the distance calculation unit 510. 

With reference to the position of the camera (image input units 100 and 101) at time to, the object distance Is 
calculated as: 



10 Similarly, moving amounts and By are added to coordinates and Yj, of the object in the X and Y directions to 

obtain corrected coordinates Xj,' and Yj/. 

The number of obtained coordinates of object points increases with time t. By performing an image sensing operation 
as shown Fig. 5. detailed distance information in a wide range can be obtained. 

The true object distance data obtained by the object distance coupling unit 500 is sent to the image interpolation 
IS unit 3605 and the image transformation unit 3608. 

The image Interpolation unit 3605 generates an image viewed from a viewpoint based on reproduction order data 
(equivalent to the viewpoint locus data 330) by using the interpolation processing method described In the first embod- 
iment. This interpolation method is different from that in the first embodiment in that corresponding points input from the 
corresponding point input unit 81 0 are directly used, and the remaining points are obtained by the above block matching 
20 method. 

The image transformation unit 3608 performs rotational transformation of the image generated by the image inter- 
polation unit 3605 on the basis of the above viewpoint direction data. This method Is the same as the rotational trans- 
formation method described in the first embodiment, and hence a description thereof will be omitted. 

The image transformed on the basis of the viewpoint direction is displayed on the display 2004 by driving the display 
25 i/F 2003 through the image reproduction unit 3606. 

Fig. 34 shows an example of the arrangement of the corresponding point Input unit 810 in Fig. 30. 

Image data sent from the Image input units 100 and 101 are delayed by delay elements 8401 , 8402, and 8403 in 
units of frames and sent to the display device 811 to be displayed on fourth image 8321 , respectively. 

As described above with reference to Fig. 31 , corresponding point coordinates are input by using the pointing device 
30 812. The acquired corresponding point coordinate data Is sent to the moving amount calculation unit 820. The repre- 
sentative movement vector (B^^.By.B^) is then generated. 

As has been described above, according to this embodiment, the corresponding point designating means is used 
to obtain corresponding points on Images. On the basis of the corresponding points, the representative movement vector 
of each Image input unit (camera) is calculated to generate distance data at a higher resolution. In addition, an interpo- 
35 lated image at a higher resolution can be generated. Consequently, accurate Image reproduction can be realized. 

Note that each processing unit described above Is constituted by a corresponding processing program, which is 
stored in the ROM 2002 in advance. The MRU 2000 reads out and interprets these programs, and executes them, 
thereby executing the above processing in each processing unit described above. 

The present invention can be applied to a system constituted by a plurality of devices or to an apparatus comprising 
40 a single device. 

Furthermore, the invention Is applicable also to a case where the invention is embodied by supplying a program to 
a system or apparatus. In this case, a storage medium, storing a program according to the Invention constitutes the 
Invention. The system or apparatus Installed with the program read from the medium realizes the functions according 
to the invention. 

45 As many apparently widely different embodiments of the present invention can be made without departing from the 

spirit and scope thereof, it Is to be understood that the invention Is not limited to the specific embodiments thereof except 
as defined in the appended claims. 



so Claims 

1. An image sensing apparatus characterized by comprising: 
image input means (100) for Inputting an image; 

detection means (200) for detecting a state of movement of said image input means in an Image input oper- 

55 ation; 

storage means (310) for storing a plurality of images Input by said image Input means and movement Infor- 
mation corresponding to the state of movement detected by said detection means; and 

Image generating means (320) for generating an image from an arbitrary viewpoint position on the basis of 
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10 



the plurality of Images and the movement Infomnatlon stored In said storage means. 

2. The apparatus according to claim 1, wherein said image input means outputs a plurality of images obtained by 
sensing the image from relatively different positions, and stores the images in said storage means. 

3. The apparatus according to claim 1, wherein said image input means includes a multi-eye image sensing system 
having a plurality of image sensing systems. 

4. The apparatus according to claim 1 , wherein said image input means includes a single-eye image sensing system. 

5. The apparatus according to claim 1 , wherein said image generating means generates an image from the arbitrary 
viewpoint position by interpolating the plurality of images on the basis of the movement information. 

6. The apparatus according to claim 5, further comprising viewpoint position input means for Inputting coordinates of 
IS the arbitrary viewpoint position. 

7. The apparatus according to claim 6, wherein said viewpoint position input means inputs a permutation of coordinates 
of arbitrary viewpoint positions in correspondence with a reproduction order. 

20 8. The apparatus according to claim 6, wherein said viewpoint position input means further comprises display means 
for displaying a relationship between a range of Images sensed by said image input means and time points of image 
sensing, and 

Inputs coordinates of a viewpoint position on the basis of the relationship displayed on said display means. 

2S 9. The apparatus according to claim 7. wherein said viewpoint position Input means further comprises display means 
for displaying a relationship between a range of Images sensed by said Image Input means and time points of image 
sensing, and 

inputs coordinates of a viewpoint position on the basis of the relationship displayed on said display means. 

30 10. The apparatus according to any one of claim 6, wherein said viewpoint position input means also inputs a viewpoint 
direction at the coordinates of the input arbitrary viewpoint position. 

11. The apparatus according to any one of claim 7, wherein said viewpoint position input means also Inputs a viewpoint 
direction at the coordinates of the input arbitrary viewpoint position. 

35 

12. The apparatus according to any one of claim 6, wherein said image generating means comprises: 

selection means for selecting a plurality of images sensed at viewpoint positions near a viewpoint position 
Input by said viewpoint position input means; 

corresponding point detection means for dividing the plurality of selected images into predetermined blocks, 
40 and detecting corresponding points on the basis of block matching: 

calculation means for calculating coordinates including a distance to an object to be sensed on the basis of 
the detected corresponding points; and 

image generating means for generating an interpolated Image corresponding to a viewpoint position input by 
said viewpoint position input means on the basis of the corresponding points, the coordinates of the object, and the 
"fs movement Information. 

13. The apparatus according to any one of claim 7, wherein said image generating means comprises: 

selection means for selecting a plurality of images sensed at viewpoint positions near a viewpoint position 
Input by said viewpoint position Input means; 
so corresponding point detection means for dividing the plurality of selected Images Into predetermined blocks, 

and detecting corresponding points on the basis of block matching; 

calculation means for calculating coordinates including a distance to an object to be sensed on the basis of 
the detected corresponding points; and 

Image generating means for generating an interpolated Image corresponding to a viewpoint position input by 
ss said viewpoint position Input means on the basis of the corresjaonding points, the coordinates of the object, and the 

movement Information. 

14. The apparatus according to claim 1, wherein said corresponding point detection means comprises: 
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first corresponding point detection means for detecting, by block nnatcliing, corresponding points between a 
plurality of images which are sensed from different viewpoints at the same time; and 

second corresponding point determination means for performing second corresponding point detection 
between corresponding points detected by said first detection means at first and second times upon determining 
s that an evaluation function including a predetermined feature amount of the corresponding points at the first and 

second times and the movement information is smaller than a predetermined threshold. 

15. The apparatus according to claim 14, wherein the feature amount is a density. 

10 16, The apparatus according to claim 14, wherein the feature amount includes basic color components. 

17. The apparatus according to claim 14, wherein the evaluation function is a function of a distance between a feature 
amount of corresponding points at the first time and a feature amount of positions obtained by correcting positions 
of corresponding points at the second time in accordance with the movement information. 

IS 

18. The apparatus according to claim 14, wherein the threshold is set to be larger with an increase in amount of the 

movement information. 

19. The apparatus according to claim 14, wherein the threshold is set to be larger as the object is located closer to said 
20 Input means. 

20. The apparatus according to claim 1, wherein said image generating means further comprises rotational transfor- 
mation means for performing rotational transformation of an image generated by said image generating means on 
the basis of a viewpoint direction Input by said viewpoint position input means. 

25 

21. The apparatus according to claim 20, wherein said image generating means further comprises: 

back-and-forth direction moving amount input means for inputting a moving amount of a viewpoint in a 
back-and-forth direction; and 

back-and-forth movement transformation means for performing back-and-forth movement transformation of 
30 an image having undergone rotational transformation in said rotational transformation means on the basis of the 

input moving amount of the viewpoint in the back-and-forth direction. 

22. The apparatus according to claim 1 2, wherein said image generating means further comprises: 

back-and-forth direction moving amount input means for Inputting a moving amount of a viewpoint in a 
3S back-and-forth direction; and 

back-and-forth movement transformation means for performing back-and-forth movement transformation of 
an image generated by said image generating means on the basis of the input moving amount of the viewpoint in 
the back-and-forth direction. 

40 23. The apparatus according to claim 1 3, wherein said image generating means further comprises: 

back-and-forth direction moving amount input means for inputting a moving amount of a viewpoint in a 
back-and-forth direction; and 

back-and-forth movement transformation means for performing back-and-forth movement transformation of 
an image generated by said image generating means on the basis of the input moving amount of the viewpoint in 
45 the back-and-forth direction. 

24. The apparatus according to claim 3, wherein said corresponding point detection means comprises: 

stereo image corresponding point detection means for detecting, by block matching, corresponding points 
between stereo images which are sensed from different viewpoints at the same time; 

^0 left image corresponding point detection means for detecting corresponding points on a left image of the stereo 

images detected by said stereo image corresponding point detection means at a first time, and corresponding points 
on a left Image of the stereo Image at a second time upon determining that a value of an evaluation information 
including a predetermined feature amount of corresponding points on the left image of the stereo Images at the first 
and second times and the movement information is smaller than a predetermined threshold; and 

5S right image corresponding point detection means for detecting corresponding points on a right image of the 

stereo images detected by said stereo image corresponding point detection means at the first time, and correspond- 
ing points on a right Image of the stereo image at the second time upon determining that a value of an evaluation 
information including a predetermined feature amount of corresponding points on the right image of the stereo 
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images at the first and second times and the movement information is smaller than a predetermined threshold. 

25. The apparatus according to claim 3, wherein said calculation means calculates a first coordinate value including a 
distance to the object on the basis of corresponding points between the stereo images detected by said correspond- 

s ing point detection means at the first time. 

calculates a second coordinate value including a distance to the object on the basis of corresponding points 
between the stereo images detected by said corresponding point detection means at the second time, 

calculates a third coordinate value Including a distance to the object on the basis of corresponding points on 
the left image detected by said corresponding point detection means, 
^0 calculates a fourth coordinate value including a distance to the object on the basis of corresponding points on 

the right Image detected by said corresponding point detection means, and 

sets the first to fourth coordinate values as coordinate values of the object if the movement information is 
larger than a baseline length of said Image Input means, and the first and second coordinate values as coordinate 
values of the object if the movement information is smaller than the baseline length of said image input means. 

IS 

26. The apparatus according to claim 1 , wherein the movement information is a movement vector in an image input 
operation performed by said image input means. 

27. The apparatus according to claim 1 , wherein said apparatus comprises reproduction means for displaying an image 
20 generated by said image generating means on a monitor, and 

said image generating means interpolates an Image corresponding to a viewpoint position and a viewpoint 
direction of a user within a screen of the monitor on the basis of pieces of image information stored in said storage 
means, and outputs the interpolated image to said reproduction means. 

25 28. The apparatus according to claim 1 , further comprising reproduction order setting means for setting an order in 
which images are reproduced by said reproduction means. 

29. The apparatus according to claim 26, wherein said detection means comprises: 

corresponding point input means for inputting corresponding points on a plurality of images input by said Input 
30 means; and 

representative movement vector generating means for generating a representative movement vector from 
each movement vector based on each of the corresponding points. 

30. The apparatus according to claim 26, wherein said detection means comprises: 

35 corresponding point detection means for detecting corresponding points on a plurality of images from said 

input means by block matching; and 

representative movement vector generating means for generating a representative movement vector from 
each movement vector based on each of the corresponding points. 

40 31 . An image sensing method characterized by comprising: 

image input step for inputting an image using image input means ; 

detection step for detecting a state of movement of said image input means in an image input operation; 

storage step for storing a plurality of images input at said image input step and movement information corre- 
sponding to the state of movement detected by said detection step; and 
45 image generating step for generating an image from an arbitrary viewpoint position on the basis of the plurality 

of images and the movement infomnation stored in said storage step. 

32. The method according to claim 31 , wherein said image input step includes a step for outputting a plurality of images 
obtained by sensing the image from relatively different positions. 

so 

33. The method according to claim 31 , wherein said image input step makes use of a multi-eye image sensing system 
having a plurality of image sensing systems. 

34. The method according to claim 31 , wherein said image input step make use of a single-eye image sensing system 
55 having one image sensing system. 

35. The method according to claim 31, wherein at said image generating step an image from the arbitrary viewpoint 
position by interpolating the plurality of images on the basis of the movement infonnation is generated. 
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36. The method according to claim 35, further comprising viewpoint position Input step for inputting coordinates of the 
arbitrary viewpoint position. 

37. The method according to claim 35, wherein at said viewpoint position Input step a permutation of coordinates of 
arbitrary viewpoint positions In correspondence with a reproduction order is input. 

38. The method according to claim 36, wherein said viewpoint position input step further comprises display step for 
displaying a relationship between a range of images sensed by said Image input step and time points of Image 
sensing, and 

inputting coordinates of a viewpoint position on the basis of the relationship displayed on said display step. 

39. The method according to claim 37, wherein said viewpoint position input step further comprises display step for 
displaying a relationship between a range of images sensed by said image input step and time points of Image 
sensing, and 

inputting coordinates of a viewpoint position on the basis of the relationship displayed on said display step. 

40. The method according to any one of claims 36, wherein at said viewpoint position input step a viewpoint direction 
at the coordinates of the input arbitrary viewpoint position is also Input. 

41. The method according to any one of claims 37, wherein at said viewpoint position Input step a viewpoint direction 
at the coordinates of the Input arbitrary viewpoint position is also input. 

42. The method according to any one of claims 36, wherein said image generating step comprises: 

selection step for selecting a plurality of images sensed at viewpoint positions near a viewpoint position input 
by said viewpoint position input step; 

corresponding point detection step for dividing the plurality of selected images into predetermined blocks, and 
detecting corresponding points on the basis of block matching; 

calculation step for calculating coordinates including a distance to an object to be sensed on the basis of the 
detected corresponding points; and 

image generating step for generating an interpolated image corresponding to a viewpoint position input by 
said viewpoint position input step on the basis of the corresponding points, the coordinates of the object, and the 
movement information. 

43. The method according to any one of claims 37, wherein said image generating step comprises: 

selection step for selecting a plurality of images sensed at viewpoint positions near a viewpoint position input 
by said viewpoint position Input step; 

corresponding point detection step for dividing the plurality of selected images into predetermined blocks, and 
detecting corresponding points on the basis of block matching; 

calculation step for calculating coordinates including a distance to an object to be sensed on the basis of the 
detected corresponding points; and 

image generating step for generating an Interpolated image corresponding to a viewpoint position input by 
said viewpoint position input step on the basis of the corresponding points, the coordinates of the object, and the 
movement information. 

44. The method according to claim 31 , wherein said corresponding point detection step comprises: 

first corresponding point detection step for detecting, by block matching, corresponding points between a 
plurality of images which are sensed from different viewpoints at the same time; and 

second corresponding point determination step for performing second corresponding point detection between 
corresponding points detected by said first detection step at first and second times upon determining that an eval- 
uation function including a predetermined feature amount of the corresponding points at the first and second times 
and the movement information is smaller than a predetermined threshold. 

45. The method according to claim 44, wherein the feature amount is a density. 

46. The method according to claim 44, wherein the feature amount includes basic color components. 

47. The method according to claim 44, wherein the evaluation function is a function of a distance between a feature 
amount of corresponding points at the first time and a feature amount of positions obtained by correcting positions 
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of cx>rrespondmg points at the second time in accordance with the movement information. 

48. The method according to claim 44, wherein the threshold is set to be larger with an Increase in amount of the 
movement information. 

5 

49. The method according to claim 44. wherein the threshold is set to be larger as the object is located closer to said 
Input step. 

50. The method according to claim 31 , wherein said image generating step further comprises rotational transformation 
10 step for performing rotational transformation of an image generated by said image generating step on the basis of 

a viewpoint direction input by said viewpoint position input step. 

51. The method according to claim 50, wherein said image generating step further comprises: 

back-and-forth direction moving amount input step for inputting a moving amount of a viewpoint in a 
IS back-and-forth direction; and 

back-and-forth movement transformation step for performing back-and-forth movement transformation of an 
image having undergone rotational transformation in said rotational transformation step on the basis of the input 
moving amount of the viewpoint in the back-and-forth direction. 

20 52. The method according to claim 42, wherein said Image generating step further comprises: 

back-and-forth direction moving amount input of inputting a moving amount of a viewpoint in a back-and-forth 

direction; and 

back-and-forth movement transformation step for performing back-and-forth movement transformation of an 
image generated by said image generating step on the basis of the input moving amount of the viewpoint in the 
2S back-and-forth direction. 

53. The method according to claim 43, wherein said image generating step further comprises: 

back-and-forth direction moving amount input of Inputting a moving amount of a viewpoint in a back-and-forth 
direction; and 

^ back-and-forth movement transformation step for performing back-and-forth movement transformation of an 

image generated by said image generating step on the basis of the input moving amount of the viewpoint in the 
back-and-forth direction. 

54. The method according to claim 33, wherein said corresponding point detection step comprises: 

55 stereo image corresponding point detection step for detecting, by block matching, corresponding points 

between stereo Images which are sensed from different viewpoints at the same time; 

left image corresponding point detection step for detecting corresponding points on a left image of the stereo 
images detected by said stereo image corresponding point detection step at a first time, and corresponding points 
on a left image of the stereo image at a second time upon determining that a value of an evaluation information 

40 Including a predetermined feature amount of corresponding points on the left image of the stereo images at the first 

and second times and the movement Information Is smaller than a predetermined threshold; and 

right image corresponding point detection step for detecting corresponding points on a right image of the 
stereo images detected by said stereo Image corresponding point detection step at the first time, and corresponding 
points on a right image of the stereo image at the second time upon determining that a value of an evaluation 
information including a predetermined feature amount of corresponding points on the right image of the stereo 
Images at the first and second times and the movement information is smaller than a predetermined threshold. 

55. The method according to claim 33, wherein at said calculation step a first coordinate value Including a distance to 
the object on the basis of corresponding points between the stereo images detected by said corresponding point 

so detection step at the first time is calculated; . 

a second coordinate value including a distance to the object on the basis of corresponding points between 
the stereo images detected by said corresponding point detection step at the second time is calculated; 

a third coordinate value including a distance to the object on the basis of corresponding points on the left 
image detected by said corresponding point detection step is calculated, 
55 a fourth coordinate value including a distance to the object on the basis of corresponding points on the right 

image detected by said corresponding point detection step Is calculated; and 

the first to fourth coordinate values as coordinate values of the object is set if the movement information is 
larger than a baseline length of said image Input step, and the first and second coordinate values as coordinate 
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values of the object if the movement information is smaller than the baseline length of said image Input step. 

56. The method according to claim 31 . wherein the movement Information is a movement vector in an image Input 
operation performed by said image input step. 

5 

57. The method according to claim 31 , wherein said method comprises reproduction step for displaying an image gen- 
erated by said Image generating step on a monitor, and 

at said image generating step an Image corresponding to a viewpoint position and a viewpoint direction of a 
user within a screen of the monitor on the basis of pieces of image information stored in said storage step is inter- 
10 polated, and the interpolated image is outputted to said reproduction step. 

58. The method according to claim 31 , further comprising reproduction order setting step for setting an order in which 
images are reproduced by said reproduction step. 

IS 59. The method according to claim 56, wherein said detection step comprises: 

corresponding point input step for inputting corresponding points on a plurality of images input by said input 
step; and 

representative movement vector generating step for generating a representative movement vector from each 
movement vector based on each of the corresponding points. 

20 

60. The method according to claim 56, wherein said detection step comprises: 

corresponding point detection step for detecting corresponding points on a plurality of images from said input 
step by block matching; and 

representative movement vector generating step for generating a representative movement vector from each 
25 movement vector based on each of the corresponding points. 

61. A camera in which three-dimensional images can be obtained by combining movement data with a plurality of 
different images. 

30 
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